Image compression and video compression have been widely used to reduce required storage or bandwidth associated with the respective image and video data. The underlying image or video data often comprises multiple channels. For example, in the digital camera application, a raw image is often captured using an image sensor with Red (R), Green (G) and Blue (B) color arrays. While the RGB color provides a convenient format to represent a color image or video for capture or display applications, the three color channels have to be treated equally for storage or transmission, which may not be efficient. In the field of image and video coding, various luminance/chrominance color formats are widely used, where each color format corresponds to a luminance component and two chrominance components. For example, YUV or YCrCb color format has been used in various image and video compression standards. Since the human vision is less sensitive to the chrominance signals, the chrominance signals (i.e., U/Cb and V/Cr) can be represented at lower spatial resolution with no degradation or minor degradation on the visual quality. There are also other variations of luminance/chrominance color formats. For example, the YCoCg color format disclosed in U.S. Pat. No. 7,155,055 and the reversible YUV color format disclosed in U.S. Pat. No. 6,044,172 use simplified color transform matrices so that only integer operations are used to convert signals between the RGB and the YCoCg/Integer YUV formats. There is also a color format conversion between RGB in Bayer pattern and the YCoCg color format as disclosed in U.S. Patent Application Publication No. US2006/0083432. In typical image/video compression systems, the compression process is applied to individual luminance and chrominance channels separately.
The color conversion between a primary color format and a luminance/chrominance format involves pixel-based operations, where the conversion is represented by a set of linear equations, i.e., a conversion matrix. While the luminance/chrominance color format removes some redundancy among different channels of original image or video in the primary color format, there is still substantial amount of correlation among different channels in the luminance/chrominance color format. It is desirable to exploit the redundancy among different channels for image or video in the luminance/chrominance color format. Furthermore, it is desirable to reduce the impact on an existing image or video compression system while exploiting the redundancy among different channels.
FIG. 1 illustrates an example of image compression system such as a digital camera. Image sensor 110 is used to capture raw image data 120. Image compressor 130 receives raw image data 120 and applies image compression to generate compressed image data 140. Usually, image compression is applied to image data in a luminance/chrominance color format, such as YUV. Therefore, color conversion is performed in image compressor 130 to convert the raw image into the luminance/chrominance color format first and image compression is then applied to individual channels. FIG. 2 illustrates an example of image compression for transmission or interface applications such as MIPI (Mobile Industry Processor Interface), DSI (Display Serial Interface) and HDMI (High-Definition Multimedia Interface), Wi-Fi Display or Bluetooth. Block diagram 200A corresponds to the transmitter side, where raw image data 220 is received from transmitter 210. In the example of HDMI interface between a set-top box and an HDTV set, the transmitter corresponds to the set-top box and the receiver corresponds to the HDTV set. Image compressor 230 compresses the raw image data into compressed image data 240. The compressed image data is then transmitted over the media such as a physical cable or wireless link. Block diagram 220B corresponds to the receiver side, where compressed data 250 is received over a media. The compressed data is then decompressed by image decompressor 260 to recover raw image data 270. The recovered raw image data is then provided to the receiver for display.
FIG. 3 illustrates exemplary block diagrams of image compression systems. System 300A corresponds to a compression system based on spatial prediction. Original image 310 with three channels is processed by quantization. The original image may correspond to color pixels in the RGB format. The original image is converted to a luminance/chrominance color format such as YUV using a 3×3 conversion matrix. After matrix conversion, the result is quantized by input quantization 320 to a fixed data depth, such as 8 bits or 10 bits. Spatial prediction 330A is then applied to individual channels to generate residue data, which is then compressed by entropy coding 350A to generate compressed data. Spatial prediction is well known in the field of image and video coding. For example, the image can be divided into blocks and each block can be predicted based on neighboring blocks that have been processed. System 300B corresponds to a compression system based on frequency transform, where frequency transform 340B is applied to quantized image data. For example, discrete cosine transform (DCT) can be applied to individual channels. The transform coefficients are often quantized (the quantizer for transform coefficients is not explicitly shown in system 330B) and then coded by entropy coding 350B to generate compressed image data. The quantizer for the frequency transformed data is often used for the purpose of bit rate control. System 300C illustrates a compression system based on both spatial prediction and frequency transform, where frequency transform 340C is applied to residual signal from spatial prediction 330C. The transform coefficients are quantized (the quantizer for transform coefficients is not explicitly shown in system 330C) and then coded by entropy coding 350C to generate compressed image data. In FIG. 3, the residual signals are generated by spatial processing 330A, frequency transform 340B or both spatial prediction 330C and frequency transform 340C. Systems shown in FIG. 3 are intended to illustrate examples of generating residual signals in an image compression system. There are many other means to generate residual signals for image compression. For example, the image compression system may use wavelet or pyramid techniques to form residual signals. The invention disclosed herein is not limited to the image compression systems shown in FIG. 3.
FIG. 4 illustrates an exemplary block diagram of video compression system, where the system consists of three source data channel (410A-410C) corresponding to three input channels. The source data channels are predicted by respective Intra/Inter prediction from block 420. The residual signals (414A-414C) correspond to the differences between source data channels and respective prediction using subtractors 412A-412C. The Intra/Inter prediction process is well known in the art. For example, in the MPEG-2, H.264/AVC and the emerging High Efficiency Video Coding (HEVC) coding standards, a form of Intra/Inter prediction is used. The Intra prediction and Inter prediction is often adaptively selected for each image area such as a macroblock, a largest coding unit (LCU), a slice, or a picture. The Inter prediction may include motion estimation to exploit temporal redundancy among images in the video sequence. While the redundancy within each input channel has been reduced by the residue generation process, the residual signals (414A-414C) may not be in a form suited for entropy coding. Usually the residual signals are subject to further processing to convert the signal into symbols suited for entropy coding. For example, the residual signals are further processed by two-dimensional transform and quantization (T/Q, 450A-450C) as shown in FIG. 4. Such further processing for the encoder side is referred to as coding process in this disclosure. Accordingly, the two-dimensional transform, quantization, or a combination of both two-dimensional transform and quantization is an example of coding process. The frequency transform used for image coding, such as DCT, can be used as the two-dimensional transform for video compression. Other two-dimensional transforms, such as integer transform, may also be used to achieve high computational efficiency for video applications. The quantized transform coefficients are then compressed using entropy coder 470 to generate compressed video output. The entropy coder usually compressed data symbols into compressed bitstream. For example, the data symbols correspond to the quantized transform coefficients. It is well known in the art that the data symbols may also include other coding information.
The encoder side also needs to reconstruct the coded video in order to form prediction. Accordingly, inverse coding process has to be applied to the data symbols to recover the residual signals. For example, the quantized transform coefficients from T/Q 450A-450C are processed by inverse quantization and inverse two-dimensional transform (IQ/IT, 460A-460C) to generate recovered residual signals (444A-444C). The IQ/IT processed signals are added to the previous prediction signals from Intra/Inter prediction 420 to form reconstructed signals using adders 442A-442C. The reconstructed signals are then processed by in-loop filters 440A-440C to reduce coding artifacts before the reconstructed signals are stored in frame buffers 430A-430C for later use by Intra/Inter prediction 420. FIG. 4 illustrates an example of reconstruction loop in the encoder side to reconstruct the image or video data based on residual signals, where the reconstruction loop comprises adders (442A-442C), in-loop filters (440A-440C), frame buffers (430A-430C) and reconstruction using Intra/Inter prediction (part of block 420). However, a coding system may use different system partition. For example, the inverse quantization may also be included in the reconstruction loop in the encoder side to reconstruct the image or video data based on residual signals. In FIG. 4, the residual signals are generated from the source data channels and Intra/Inter prediction using subtractors 412A-412C. The residual signals are reproduced after T/Q 450A-450C and IQ/IT 460A-460C processing. There are many other means to generate residual signals for video compression. The invention to be disclosed hereinafter is not limited to the video compression systems shown in FIG. 4.